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Redundant  disk  arrays  are  single  fault  tolerant,  incorpcMating  a  layer  of  error  handling  not  found  in  non- 
redundant  disk  systems.  Recovery  from  these  erras  is  complex,  due  in  part  to  the  laige  number  of  erro¬ 
neous  states  the  system  may  reach.  The  established  approach  tt>  error  recovery  in  disk  systems  is  to 
transition  directly  from  an  erroneous  state  to  completion.  This  technique,  known  as  forward  error  recov¬ 
ery,  relies  upon  the  context  in  which  an  error  occurs  to  determine  the  steps  required  to  reach  completion, 
which  impl^  forward  etior  recovery  is  design  specific.  Forward  error  recovery  requires  the  enumera¬ 
tion  of  all  erroneous  states  the  system  may  reach  and  the  construction  of  a  forward  path  from  each  erro¬ 
neous  stale.  We  propose  a  method  of  error  recovery  which  does  not  rely  upon  the  enumeration  of 
oroneous  states  or  the  context  in  which  errors  occur.  When  an  error  is  encountered,  we  advocate  mech¬ 
anized  rtccNtty  to  an  error-free  stale  6om  which  an  opnation  may  be  retried.  Using  a  form  of  back- 
wad  error  recovery,  we  are  able  to  manage  the  complexity  of  error  recovery  in  redundant  disk  arrays 
without  sacrificing  perftnmance. 
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1.  Introduction 


Through  the  use  of  redundancy,  disk  arrays  can  be  made  single  fault  lolerani  Disk  arrays  vshich  pros  idc  single 
fault  tolerance,  categonzed  by  a  taxonomy  known  as  RAID  (Rcdundani  Arrays  ot  Inexpensive  Disksi  m  I9KK 
and  summanzed  in  Figure  1.  have  become  a  important  class  of  storage  devices  IPaltersonHHI  As  such  many 
companies  are  now  engaged  in  the  design  of  RAID  systems  While  many  nl  ihese  companies  possess  a  tremcn 
^  dous  amount  ot  experience  developing  nonredundani  disk  systems.  RAID  systems  introduce  .1  new  ilesign 

challenge  isolating,  at  the  level  of  the  redundant  disk  array  control,  the  ettecis  ot  single  taults  iherebs  irans 
parentis  handling  a  large  cla.vs  of  errors  Figure  2  shows  the  siru>.ture  ot  a  redundant  disk  anas  ..onirol  ssstem 
This  new  error  recovery  task  is  made  particularly  dithcult  siikc  an  important  feature  ot  disk  anays  is  high  ^on 
•  currency,  rnaking  the  number  ot  erroneous  states  whn.h  the  ssstem  may  reach  uncountable 
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Figure  I  RAID  Leeds  0,  I,  3,  and  5.  This  figure  depicts  the  data  layout  and  redundancy  organizations  tor  the 
most  prevalent  RAID  levels  using  an  array  ot  four  disks.  Data  units  represent  the  unit  of  data  access  supponed 
by  the  array  Panty  units  represent  redundancy  information  generated  from  the  bitwise  exclusive-or  (parity )  ot 
a  collection  of  dau  units.  The  redundancy  group  formed  by  a  parity  unit  and  the  data  units  it  protects  is 
commonly  known  as  a  panty  grvup. 


RAID  level  0  offers  no  redundancy  so  it  is  not  single  fault  tolerant.  In  this  illustration,  data  is  block  interleaved 
meaning  that  the  array  is  optimized  for  small  transfer  sizes  with  each  request  being  serviced  by  a  single  drive, 
increasing  throughput.  Data  may  also  be  bit  interleaved,  optimizing  performance  for  large  transfer  si^.es  by 
using  every  drive  to  transfer  data  in  parallel,  increasing  bandwidth. 

RAID  level  I  offers  the  simplest  form  of  redundancy,  maintaining  two  copies  of  each  data  unit,  each  stored  on 
a  different  disk.  Also  referred  to  as  mirroring,  this  method  of  redundancy  has  the  highest  capacity  overhead  ot 
the  RAID  architectures:  50%  of  disk  space  is  consumed  by  redundant  information.  Because  the  redundant 
information  is  a  simple  copy  and  both  copies  reside  on  independent  disks,  read  operations  can  be  serviced  by 
either  copy,  making  balancing  the  read  workload  across  the  array  easier. 


RAID  levei  3  is  bit  interleaved  to  optimize  bandwidth  and  relies  on  parity-based  redundancy  to  reduce 
capacity  overhead.  Redundancy  is  maintained  on  a  single  drive,  the  parity  drive.  Parity  is  calculated  as  the 
biiwiie  exclusive-or  of  the  dau  drives. 
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Fifure  2  Abttractlow  tai  tkc  Storage  Hierarchy.  Shown  are  four  layers  of  abstraction  found  in  a  typical 
storage  hierarchy  and  their  relationship.  Disk  drives  provide  durable  storage  and  implement  read  and  write 
actions.  Redundant  disk  array  control  systems  increase  the  reliability  and  availability  of  disk  storage  ant< 
impleineal  read  and  write  operations  as  a  composition  of  disk  actions.  Filesystem  and  database  abstractions 
provide  more  natural  interfaces  to  durable  storage,  implementing  operations  such  as  tile  open,  delete  tile,  add 
record,  and  read-modtfy-wnte.  Finally,  applies  ons  run  as  a  part  of  the  user  process  and  provide  an  interface  to 
the  outside  world,  implementing  loois  such  as  word  processors,  simulators,  data  visualization,  and  acciHinting 
systems 

Disk  drives  are  not  fault  tolerant,  meaning  that  any  er  .  ‘  ouniercd  during  a  disk  action  results  in  loss  ot 
dau  Redundant  disk  arrays  make  disk  storage  single  ;  >  t.  guaranteeing  no  loss  of  dau  and  successful 

completion  of  operations  when  a  single  disk  failure  occw.  ■  .  s  due  lo  the  existence  cf  multiple  disk  failures 

may  result  in  loss  of  data  in  the  array.  Rlesystems  generali,.  do  provide  additional  fault  tolerance,  but  do 
attempt  to  minimize  the  effects  of  dau  loss  presented  (o  an  vP^ication  (LefflerMOl  Databases,  however, 
usually  do  provide  higher  levels  of  fault  tolerance;  in  particular,  operations  are  generally  atomic,  meaning  that 
iransnctiotts  which  fail  leave  the  applicalions's  view  of  the  system  ur/'hsnged 


The  traditional  approach  to  recovering  from  errors  in  RAID  systems  has  been  to  transition  dl*ecily  from  an 
erroneous  state  to  a  completion  state.  This  approach,  known  as  forward  error  recovery,  relies  upon  ;hc  enumer¬ 
ation  of  each  erroneous  state  the  system  may  reach.  Since  this  is  a  large  number,  the  usk  of  guar^jueeing  cor¬ 
rect  error  recovery  while  maintaining  high  concurrency  can  quickly  become  overwhelming. 

Another  approach,  backward  error  recovery,  eliminates  the  complexity  associated  with  ideniirying  all  errone¬ 
ous  stales  and  transitions  from  (hem.  This  is  accomplished  by  saving  ihe  state  of  the  system  prior  us  each  oper¬ 
ation  and  then  restoring  this  state  if  an  operation  fails.  Once  restoration  is  compieic.  ihc  system  is  free  fmn' 
error  and  processing  resumes.  Unfortunately,  systems  employing  backward  error  recovery  can  expect  degraded 
performance  due  to  the  cost  associated  with  saving  the  sute  ot  the  system  prior  to  each  operation.  In  some 
cases,  such  as  write  operations,  the  amount  of  work  required  to  save  sute  information  is  comparable  to  the 
requested  work,  leading  to  a  large  degradation  in  performance 

We  propose  a  tnethod  of  error  recovery  based  upon  backward  error  recovery,  but  do  not  degrade  performance 
by  saving  additional  state  information  dunng  normal  processing.  When  an  error  is  encountered,  recovery  is 
p^ormed  to  a  convenient  error-free  state,  not  necessarily  the  same  sute  the  system  was  in  at  the  beginning  of 
the  operation.  Once  the  system  is  in  an  error-ffec  sute,  operation  resumes  and  the  operation  which  encountered 
the  error  is  retried.  This  approach  eliminates  the  complexity  of  transitioning  from  an  erroneous  state  to  a  com¬ 
pletion  state.  Furthermore,  because  we  do  not  require  recovery  to  a  previously  existing  sute.  we  do  not  incur 
performance  degradation  normally  associated  with  backward  error  recovery. 

The  remainder  of  this  paper  demonstrates  the  problem  of  error  recovery  in  redundant  disk  arrays,  current  solu¬ 
tions  and  their  shortcomings,  and  concludes  with  our  approach  to  this  problem.  Section  2.  discusses  the  moti¬ 
vations  for  our  interest  in  this  problem.  These  include  a  description  of  the  complexity  of  error  recovery  in 
redundant  disk  arrays,  the  shortcomings  of  the  current  approach  to  error  recovery,  and  the  benefits  of  pursuing 
a  better  approach.  In  Section  3..  we  examine  other  approaches  to  error  recovery  and  their  weaknesses.  Section 
4.  presents  our  approach  to  error  recovery,  which  manages  complexity  and  enables  exploration  of  the  disk 
array  design  space  without  degrading  performance  overhead  during  normal  processing.  Rnaily.  in  Section  S.. 
we  conclude  with  a  summary  of  the  benefits  of  our  approach  and  a  brief  review  of  work  in  progress  to  demon¬ 
strate  this  approach  as  valid. 


Pax»  4  Ilf  IH 


2.  Motivation 


2.1.  Motivations  for  Redundant  Disk  Arrays 


Disk  arrays  are  a  well  established  method  of  using  parallelism  to  reduce  response  time  in  disk  storage  systems 

*  [Kim86,  Salem86,  Kat289,  Reddy 89].  Response  time  is  the  total  amount  of  time  required  to  service  a  request 
made  to  a  disk  system  and  is  composed  of  three  components:  queueing  time,  the  time  a  request  spends  in  a 
queue  waiting  to  begin  execution;  positioning  time,  the  time  required  to  position  the  disk  head  to  useful  data; 
and  tranrfer  time,  the  time  required  to  transfer  data  to  or  from  the  disk.  Queueing  time  is  reduced  when  multi- 

*  pie  requests  are  serviced  concurrently  and  transfer  time  is  reduced  by  transferring  data  from  disks  in  parallel. 

Simple  disk  systems  are  not  fault  tolerant,  a  single  fault  can  lead  to  data  loss.  The  accepted  failure  model  of 
such  nonredundant  disk  systems  requires  only  error  detection,  the  recognition  of  the  presence  of  an  error 
{Lampson79].  This  is  acceptable  since  applications  which  require  fault  tolerance  implement  schemes  to  sur¬ 
vive  data  loss  at  the  application  level  of  the  system  in  the  following  way.  After  the  detection  of  an  error,  the 
disk  system  notifies  a  client  who  is  responsible  for:  damage  confinement  and  assessment,  the  understanding 
and  limitation  of  the  effects  of  the  detected  error;  error  recovery,  the  removal  of  the  effects  of  the  detected 
error  from  the  system;  and  fault  treatment,  isolating  the  fault  which  caused  the  detected  error  and  removing  it 

from  the  system.' 

Disk  systems,  particularly  when  configured  as  arrays,  may  be  composed  of  large  numbers  of  disks.  As  the 
number  of  disks  in  the  array  increases,  reliability,  the  probability  that  the  disk  array  will  function  correctly,  and 
availability,  the  probability  that  the  disk  array  is  able  to  service  requests,  may  decrease  to  unacceptable  levels 
since  dau  loss  occurs  on  the  first  failure  (Gibson93|.  This  problem  may  be  further  aggravated  because,  as 
Patterson.  Gibson,  and  Kau  suggest,  commodity  disks  may  be  employed  in  order  to  reduce  the  cost  of  storage 
in  the  array  (Patterson88). 

Because  reliability  and  availability  are  critical  chiuacteristics  of  storage  systems,  disk  arrays  are  usually 
designed  to  be  single  fault  tolerant.  This  is  accomplished  by  storing  redundant  data  in  the  disk  array  |Gib- 
son89,  Gibson92].  Instead  of  propagating  errors  resulting  from  a  disk  failure  to  a  client  to  handle,  the  redun¬ 
dant  disk  array  now  performs  recovery  from  these  errors,  hiding  their  effects  from  clients  and  providing 
continuous  service  throughout  the  life  of  the  fault. 


2.2.  Error  Recovery  is  Complex 

Redundant  disk  arrays  are  required  to  provide  service  from  two  distinct  operating  states:  normal  and  degraded 
The  vray  exists  in  a  normal  operating  state  when  no  faults  are  present.  Because  redundant  disk  arrays  are  sin¬ 
gle  fault  tolerant,  they  are  also  expected  to  provide  service  in  a  degraded  operating  state  which  exists  when  a 
single  fault,  in  our  case  a  disk  failure.  ^  is  present  in  the  system.  When  two  or  more  faults  exist,  the  array  may 
be  in  a  failed  operating  state  in  which  data  is  lost  and  service  discontinued. 

When  an  error  is  encountered,  the  system  enters  an  erroneous  state,  meaning  that  the  physical  array  stale, 
“containing  a  failed  disk”,  is  inconsistent  with  the  state  of  the  system  as  perceived  by  operations  initialed  prior 
to  the  error.  Recovery  from  errors  requires  the  array  to  be  restored  to  a  consistent  state,  a  state  free  from  error, 
and  the  successful  completion  of  operation(s)  in-flight  at  the  time  of  the  error. 


» 


1 .  The  definttions  presented  here  are  consistent  with  those  of  the  IEEE  Technical  Comminee  on  Fault  Tolerant 
ConifMting  |Melliar-Smith77.  Anderson82.  Lee82). 

2.  Other  faults,  such  as  loss  of  power,  mechanical  failure  of  cabling,  can  be  converted  into  independent  single 
faults  in  “otthogonal"  redundant  disk  arrays  |Gibson93|. 
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Figure  3  Work  and  Resources  arc  a  Functloa  of  Current  Operating  State.  This  figure  compares  the  work 
and  resources  required  for  a  RAID  level  3  small-write  operations  in  the  normal  and  degraded  operating  states. 
Figure  3(a)  shows  the  normal  case  while  Figure  3(b)  illustrates  the  case  when  a  data  drive  has  failed.  A  RAID 
level  5  array  uses  parity-based  redundancy  to  achieve  single  fault  tolerance  and  requires  that  all  write 
operations  update  the  parity  which  protects  the  data  being  written.  When  the  array  is  in  the  normal  state,  the 
update  is  computed  as  the  exclusive-or  of  the  data  to  be  written,  the  new  data,  and  the  data  being  overwritten, 
old  data.  New  parity  is  then  generated  as  a  result  of  the  exclusive-or  of  the  old  parity  and  the  update. 

When  the  drive  which  contains  the  data  to  be  written  has  failed,  writes  must  be  performed  in  a  different 
manner.  Write  operations  to  a  failed  data  drive  are  not  able  to  write  new  data  but  are  still  able  to  write  parity. 
The  write  is  accomplished  by  reading  surviving  data,  exclusive-or' ing  this  with  the  new  data,  and  then  writing 
the  result  as  new  parity.  The  new  data  can  retrieved  by  exclusive-or* ing  the  surviving  data  and  new  parity. 


The  process  of  error  recovery  in  redundant  disk  arrays  is  complex  for  several  reasons:  many  operations  may  be 
executed  concurrently,  operations  may  be  initiated  and  complete  in  different  operating  states,  the  array  ma> 
reach  a  large  number  of  erroneous  states,  and  recovery  of  errors  due  to  single  faults  must  be  transparent  to  cli¬ 
ent  requestors. 

First,  disk  arrays  increase  performance  by  executing  many  operations  concurrently,  requiring  careful  synchro¬ 
nization  of  shwed  resources.  Errors  require  operations  to  alter  their  usage,  and  hence  synchronization,  ot 
shared  resources  in  order  to  complete.  As  concurreiKy  increases,  the  complexity  of  synchronization  escalates 

Second,  operations  in-flight  when  an  error  occurs  are  initiated  in  the  normal  operating  state  and  should  com¬ 
plete  in  the  degraded  operating  state.  As  Figure  3  illustrates,  the  work  and  resources  required  to  complete  an 
operation  in  the  normal  sute  varies  greatly  from  the  degraded  sute.  Becau.sc  of  this,  recovery  must  dynami¬ 
cally  change  the  operation's  algorithm.  It  is  this  problem  which  makes  error  recovery  in  redundant  disk  arrays 
particularty  difficult. 

Third,  the  number  of  erroneous  stales  which  a  redundant  disk  array  operation  can  encounter,  and  be  required  to 
recover  from,  is  large.  This  is  because  operations  in  redundant  disk  arrays  perform  more  work  than  operations 
in  simpler  disk  systems.  Figiue  4  demonstrates  that  the  added  disk  work  required  to  complete  a  small-write  in 
a  RAID  level  5  disk  array  alone  creates  twenty  potential  erroneous  stales.  Allowing  multiple  coiKurrent  opera¬ 
tions  multiplies  the  number  of  erroneous  slates  the  array  may  encounter. 

Finally,  redundant  disk  arrays  often  guarantee  coHtiniums  service,  meaning  that  service  is  provided  without  « 

intem^Hion  throughout  the  life  of  a  fault.  This  requires  that  all  error  recovery  and  fault  treatment  be  perfoimed 
transparently  while  the  array  continues  to  accept  and  complete  operations  from  client  applications,  tilesystems. 
or  databases. 
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1)  RDqd  precedes  WRnq 

2)  RDqd  and  RDqp  precede  WR^p 

Figure  4(a):  Disk- Work  Ordering  Constraints 


1 )  RDqd  RDqp  WR^q  WR^p 

2)  RDqd  RDqp  WR^p  WR^q 

3)  RDqd  VVR(md  RDqp  WR^p 

4)  RDqp  RDqp  WR^d  WR^p 

5)  RDqp  RDqp  '^Rnp  '^Rnd 

Figure  4(b):  Valid  Disk- Work  Orderings 


Figure  4  Ordering  of  Disk  Work  for  RAID  Level  5  Small-Write  Operation.  The  four  disk  operatirns, 
reading  old  data  (RDqq),  reading  old  parity  (RDqp).  writing  new  data  (WR^^q),  and  writing  new  parity 
(WR|Mp),  may  execute  concurrently  as  long  as  the  constraints  in  Figure  4(a)  are  satisfied.  This  allows  five 
possible  orderings,  shown  in  Figure  4(b).  Because  the  array  is  single  fault  tolerant,  any  of  the  four  disk 
operations  is  allowed  to  fail.  When  multiplied  across  the  five  possible  orderings,  this  produces  twenty 
erroneous  states  which  the  array  must  handle.  This  is  in  stark  contrast  to  a  single-actuator  disk  system  which  is 
only  required  to  write  new  data  to  disk,  fails  in  only  one  way,  and  does  not  execute  multiple  operations 
concurrently. 


23.  Forward  Error  Recovery  is  Inadequate 


The  traditional  approach  to  error  recovery  in  disk  systems,  forward  recovery,  attempts  to  remove  an  error  by 
applying  selective  corrections  to  the  erroneous  state,  simultaneously  moving  operations  forward  to  completion 
and  bringing  the  system  to  a  consistent  state  (AndersonSl).  Construction  of  these  corrective  actions  requires 
detailed  foreknowledge  of  the  errors  which  may  occur  and  the  damage  that  they  cause.  This  requires  enumera¬ 
tion  of  all  erroneous  states  the  system  may  reach. 

A  significant  drawback  of  this  type  of  recovery  is  that  the  context  of  an  error,  information  describing  the  spe¬ 
cific  type  of  activity  which  failed,  is  required  in  order  to  determine  the  appropriate  set  of  corrective  actions  to 
take.  This  is  because  the  effects  of  an  error  arc  a  function  of  the  context  in  which  it  was  detected.  For  instance, 
in  a  RAID  level  3  array,  each  disk  contains  an  equal  fraction  of  the  data  and  redundancy  information  stored  in 
the  array.  When  a  disk  becomes  inaccessible,  some  operations  will  not  require  its  data  (their  parity  and  data  are 
stored  on  other  disks),  some  operations  will  experience  loss  of  data,  while  others  will  experience  loss  of  parity 

Figure  5  illustrates  forward  recovery  oPa  write  operation  to  a  RAID  level  5  disk  array  in  which  an  error  has 
occurred,  preventing  a  small-write  operation  from  reading  old  data.  The  array  must  detect  the  presence  of  an 
error  during  a  disk  access,  recognize  the  context  of  the  error  as  "during  read  of  old  data  in  a  small-write  opera¬ 
tion  having  read  old  parity  already,"  move  the  array  to  a  consistent  operating  state,  and  successfully  complete 
the  operation.  These  actions  must  ail  be  executed  with  the  system  in  an  erroneous  state. 

Unfortunately,  as  already  shown,  error  recovery  in  redundant  disk  arrays  is  required  to  cope  with  an  unman¬ 
ageable  number  of  erroneous  stales.  One  way  to  simplify  the  task  of  forward  error  recovery  is  to  reduce  the 
number  of  erroneous  states  the  array  may  reach.  This  involves  reducing  the  amount  of  concurrency  in  the 
array,  leading  to  the  undesirable  result  of  diminished  performaiKe.  For  instance,  one  such  reduction  would  be 
to  increase  the  number  of  ordering  constraints  given  in  Figure  4.  The  number  of  states  could  easily  be  reduced 
by  forcing  rfta  to  be  written  to  disk  before  parity  is  read  and  written.  Doing  this  eliminates  the  ability  of  the 
array  to  perform  these  in  parallel,  reducing  concurrency  and  adversely  affecting  performance. 

Another  method  of  simplifkalion  is  based  on  recognizing  that  errors  that  have  identical  recoveries  can  be 
grouped  and  handled  by  a  single  error  recovery.  Th;*^  has  the  advantage  of  reducing  the  number  of  distinct  cor¬ 
rective  procedures  which  must  be  constructed;  however,  the  task  of  identifying  all  erroneous  states  remains. 
For  example,  errors  which  a  disk  may  present  to  array  software  include:  head  crash,  seek  error,  and  electronics 
failure.  All  of  these  errors  can  be  handled  in  the  same  fashion  by  declaring  the  disk  as  “failed"  and  moving  the 
array  u>  a  degraded  operating  stale. 
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allocate  memory 
read  old  parity 
read  old  data 

exclusivt-or:  new  data,  old  data,  aid  parity 
write  new  data 
write  new  parity 
deallocate  resources 


error  Inducea  algorithm  change 


place  array  in  “degraded”  state 
discard  old  parity 
allocate  additional  memory 
read  surviving  data 

exclusive-or:  new  data,  surviving  data 
write  new  parity 
deallocate  resources 


Figure  S  Forward  Error  Recovcfry  can  be  Complex.  This  figure  provides  an  example  of  the  complexity  of 
forward  error  recovery  in  a  RAID  level  5  small-write  operation.  In  this  illustration,  the  erroneous  state 
characterized  by  the  inability  of  a  small-write  operation  to  read  old  data  has  been  reached.  To  proceed  from 
this  erroneous  state  and  complete  the  operation,  new  parity  must  still  be  written.  This  is  accomplished 
according  to  Figure  3(b),  by  exclusive-or' ing  the  surviving  data  and  the  new  data.  The  operation  must  now 
read  surviving  data  to  compute  new  parity.  To  do  this,  additional  memory  necessary  to  hold  that  larger  amount 
of  data  must  be  allocated.  Because  the  update  procedure  has  been  abandoned,  the  previously  read  “old  parity” 
is  discarded. 

It  is  important  to  note  that  the  complexity  of  error  recovery  in  this  operation  is  not  limited  by  simply  knowing 
what  corrective  actions  to  take.  For  instance,  it  is  possible  that  the  “allocate  additional  memory”  action  may 
create  a  deadlock  condition.  Since  the  system  can  no  longer  accurately  predict  the  amount  of  resources  an 
operation  will  requite,  admittance  of  operations  to  the  system  based  upon  resource  scheduling  is  unreliable.  On 
the  other  hand,  pre-allocating  sufficient  resources  for  wtxst-case  error  handling  in  every  normal-mode 
operation  is  costly  and  limits  normal-mode  performance. _ 


Forward  error  recovery  must  be  designed  specifically  for  each  system.  This  is  a  result  of  the  dependence  upon 
knowledge  of  the  context  in  which  an  error  occurs  (Randell78|.  Because  of  this,  once  a  design  is  created,  it  can 
be  very  difficult  to  make  changes  to  the  design,  particularly  when  new  error  types  are  introduced  or  when 
existing  error  types  are  altered.  This  property  limits  the  scope  of  modifications  to  an  existing  code  base, 
thereby  restricting  a  designer's  ability  to  explore  the  design  space,  confining  experimentation  to  limited  depar¬ 
tures  horn  the  current  code  structure. 

Finally,  researchers  are  investigating  more  aggressive  redundant  disk  array  architectures  to  boost  performance 
(Bhide92,  Blaum94,  Cao93.  Menon93.  Stodolsky93.  Holland94|.  The  acceptance  of  these  proposals  is  put  at 
risk  due  to  their  further  increases  in  the  complexity  of  error  handling  and  the  diflkulty  of  modifying  existing 
code  structure. 

Forward  error  recovery  has  been  used  with  arguable  success  in  the  design  of  single  disk  systems  and  filesys¬ 
tems.  Single  disk  systems  are  not  fMilt  tolerant  and  do  not  execute  operations  concurrently;  hence,  error  recov¬ 
ery  is  relatively  simple.  OperatkMis  in  a  filesystem  are  complex  and  are  executed  concurrently;  however,  since 
filMystems  are  not  fault-tolerant,  errors  which  result  in  a  dau  loss  are  acceptable.  For  instance,  when  the  BSD 
4.3  UNIX  operating  system  unexpectedly  loses  access  to  a  disk,  data  may  be  lost  [Leffler90|. 
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2.4.  A  Problem  Worth  Solving 


The  demand  for  redundant  disk  arrays  is  growing  steadily.  The  value  of  RAID  systems  shipped  to  customers  is 
expected  to  be  $5.0  billion  in  1994,  reaching  Slj.O  billion  annually  by  1997.  This  compares  to  the  total  volume 
of  rigid  disks  sold,  estimated  to  be  $23.7  billion  for  1994.  Vendors  of  RAID  equipment  are  under  constant 
pressure  to  improve  performance  and  decrease  development  time.  The  difficulty  of  handling  errors  due  lo  disk 
f^ailures,  introduced  by  the  requirement  of  single  fault  tolerance,  is  a  limiting  factor  in  the  ability  of  these  com¬ 
panies  to  innovate.  Any  technology  which  alleviates  this  limitation  will  be  both  welcomed  and  encouraged. 
Our  analysis  of  error  recovery  in  redundant  disk  arrays  suggests  that  such  an  opportunity  exists. 


2.5.  Summary 


Before  continuing,  we  briefly  summarize  the  problems  we  have  observed  and  their  symptoms.  First,  redundant 
disk  arrays  mast  provide  transparent  recovery  from  errors  due  to  single  disk  failures.  This  error  recovery  is 
inherently  complex  and  difficult  to  manage,  meaning  that  implementation  is  difficult.  Second,  forward  error 
recovery,  the  traditional  approach  to  error  recovery  in  nonredundant  disk  systems,  does  not  scale  as  complexity 
is  increased,  leaving  implementors  unable  to  produce  more  aggressive  redundant  disk  array  architectures. 
Third,  the  number  of  erroneous  states  the  system  may  reach  can  be  decreased  by  restricting  concurrency 
(adversely  affecting  performance).  Fourth,  forward  error  recovery  measures  are  system  specific,  limiting  the 
ability  to  modify  existing  code  and  explore  the  design  space. 

Redundant  disk  arrays  will  always  be  required  to  recover  from  errors  due  to  single  disk  failures;  it  is,  by  defini¬ 
tion,  what  they  are  designed  to  do.  What  we  can  do  is  look  for  a  way  of  making  recovery  from  these  errors  less 
painful.  To  do  this,  a  context-free  method  of  managing  complex  error  recovery  which  does  not  degrade  perfor¬ 
mance  is  needed. 

3.  Related  Work 


3.1.  Backward  Error  Recovery 


Backward  error  recovery  removes  errors  from  a  system  by  moving  the  system  to  a  consistent  state  which 
existed  prior  to  the  introduction  of  the  error.  The  point  of  operation  that  the  system  attempts  to  reach  during 
recovery  is  known  as  a  recovery  point.  A  recovery  point  is  established  by  storing  recovery  data,  information 
which  describes  the  state  of  the  system,  as  a  part  of  normal  processing.  When  an  error  is  detected,  the  system  is 
returned  to  the  recovery  point  by  reinstating  the  recovery  data  [Randell78,  Stone89].  Previously  completed 
work  which  is  undone  as  a  result  of  moving  backward  to  a  recovery  point  must  be  redone. 

Backward  error  recovery  does  not  rely  upon  the  type  of  error  or  the  error’s  context  in  removing  the  error  from 
the  system.  Thus,  context-free  error  recovery  is  possible.  Also,  backward  error  recovery  does  not  require  enu¬ 
meration  of  all  the  erroneous  states.  This  means  that  backward  error  recovery  is  applicable  to  complex  error 
recovery  tasks.  Finally,  because  the  error  recovery  process  consists  of  saving  and  restoring  state,  independent 
of  the  error  context,  backward  error  recovery  can  be  mechanized. 

Unfortunately,  backward  error  recovery  can  be  expensive  in  terms  of  performance  and  resource  usage,  particu¬ 
larly  when  atomicity,  the  property  that  operations  either  complete  successfully  or  leave  the  system  unchanged, 
is  required.  Operations  composed  of  actions  which  guarantee  atomicity  have  particularly  simple  error  recov¬ 
ery.  By  recovering  to  the  state  which  existed  prior  to  an  operation,  backward  error  recovery  techniques  achieve 
atomicity.  As  the  amount  of  recovery  data  or  the  frequency  recovery  points  are  established  grows,  the  over¬ 
head  required  to  save  recovery  data  increases.  This  has  a  direct  impact  on  performance  since  recovery  data  is 
saved  as  a  part  of  normal  processing. 
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Figure  6  Effects  of  Inter^Process  Communication  on  Recovery.  From  [Stone89],  three  communicating 
processes  and  the  relationship  between  their  recovery  points  and  communications  are  shown  to  illustrate  the 
domino  effect.  Consider  recovery,  from  the  present  time,  of  each  of  the  three  processes.  If  Process  1  fails,  it 
recovers  to  recovery  point  1.3  and  then  continues  operation.  If  Process  2  fails,  it  recovers  to  recovery  point. 
2.3,  removing  the  effects  of  communication  with  Process  I,  requiring  Process  1  to  recover  to  it  recovery  point 
1.2.  Finally,  if  Process  3  fails,  it  recovers  to  3.3,  causing  Process  2  to  recover  to  2.2,  which  in  turn  causes 
Process  1  to  recover  to  I.l.  This  process  continues  until  processes  recover  to  their  initial  recovery  points,  1.0. 
2.0,  and  3.0. 


Finally,  as  Randell  points  out,  backward  error  recovery  in  systems  characterized  by  communicating  processes 
can  lead  to  disastrous  results  [Randell7S].  The  problem,  known  as  the  domino  effect,  occurs  when  communica¬ 
tion  has  taken  place  between  the  recovery  point  and  the  point  in  which  an  error  is  detected.  When  recovery  is 
performed,  the  effects  of  the  communication  are  undone,  requiring  recovery  of  the  other  processes  involved  in 
the  communication.  An  illustration  of  this  problem,  taken  from  [Stone89],  is  presented  as  Figure  6.  Techniques 
such  as  conversations,  which  synchronize  communicating  processes,  are  known  methods  of  avoiding  the  dom¬ 
ino  effect  [Randell75]. 

A  variety  of  backward  error  recovery  techniques  exist,  all  of  which  introduce  varying  degrees  of  overhead. 
These  techniques  fall  into  three  broad  classes:  checkpointing,  recursive  caches,  and  audit  trails  [AndersonSl]. 
We  now  examine  the  applicability  of  techniques  from  each  of  these  classes  to  the  domain  or  redundant  disk 
arrays. 


3.2.  Checkpointing 


Systems  employing  checkpointing  establish  a  recovery  point,  known  as  a  checkpoint,  by  saving  a  subset  of  the 
system  state,  known  as  checkpoint  data  [Chandy72.  Siewiorck93].  Erroneous  state  information  is  removed  by 
returning  the  system  to  a  chwkpoint  which  is  assumed  to  be  free  from  error.  The  process  of  returning  to  a 
checkpoint,  referred  to  as  rollback,  requires  the  checkpoint  data  associated  with  the  checkpoint  to  be  rein¬ 
stated.  By  returning  to  a  checkpoint,  all  work  perform^  since  the  checkpoint  is  lost  and  must  be  performed 
again. 

The  overhead  of  checkpointing  depends  upon  the  size  of  the  checkpoint  and  the  frequency  of  their  establish¬ 
ment.  The  simplest  and  least  effective  way  to  checkpoint  a  system  would  be  to  save  the  entire  state  of  the  sys¬ 
tem  at  the  start  of  each  process.  A  more  efficient  alternative  is  to  save  only  a  subset  of  the  system  state.  For 
instance,  a  technique  commonly  known  as  consistent  checkpointing  creates  process  checkpoints,  which  ore 
checkpoints  of  the  state  of  a  process  [ChandySS].  Collectively,  these  process  checkpoints  compose  a  check¬ 
point  of  the  system. 
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3  J.  Recursive  Cache 


One  solution  to  the  problem  of  large  amounts  of  recovery  data  is  the  recursive  cache,  also  known  as  a  recovery 
cache  [Homing74].  By  monitoring  actions  which  modify  the  system  state,  specific  state  information  is  saved 
in  a  recursive  cache,  prior  to  modification.  State  information  is  only  recorded  prior  to  initial  changes  from  the 
most  recent  recovery  point,  making  recursive  cache  techniques  efficient  in  the  sense  that  the  amount  of  state 
information  in  the  cache  is  minimal.  Error  recovery  is  performed  by  restoring  the  contents  of  the  recursive 
cache,  effectively  removing  modifications  of  state  and  restoring  the  system  to  the  recovery  point.  Again,  as 
records  are  restored,  all  work  which  occurred  since  their  entry  is  lost. 

Homing,  Lauer,  Melliar-Smith,  and  Randell  suggest  the  use  of  a  recursive  cache  to  implement  a  recovery 
block,  a  set  of  alternate  operations,  each  of  which  accomplishes  the  same  goal,  but  through  different  methods. 
An  acceptance  test  is  used  to  verify  correct  outcome.  When  an  alternate  fails,  state  is  restored  from  the  recov¬ 
ery  cache  and  another  alternate  is  attempted. 

A  principal  difficulty  with  the  recursive  cache  is  the  ability  to  know  what  state  changes  an  operation  will  effect 
upon  the  system  in  order  that  the  appropriate  information  may  be  entered  into  the  cache.  Even  with  this  knowl¬ 
edge,  overhead  is  still  introduced  when  saving  recovery  data. 


3.4.  Audit  TVails 


Finally,  audit  trail,  also  known  as  logging  or  journaling,  techniques  provide  the  ability  to  record  a  subset  of  the 
system  state  but,  unlike  recovery  cache  techniques,  do  not  require  foreknowledge  of  the  state  which  will  be 
changed  by  an  operation  [Bjork75,  Verhofstad78,  GraySl].  Instead,  all  changes  to  the  system  state  are 
recorded  in  stable  storage.  Recovery  is  performed  by  applying  the  inversion  of  these  records  in  LIFO  fashion, 
thereby  removing  state  changes.  As  inverted  records  are  applied,  work  is  undone.  Once  the  system  is  in  a  con¬ 
sistent  state,  some  records  may  be  applied  in  RFO  fashion  to  restore  previously  completed  work.  The  System 
R  database  recovery  manager  implements  such  an  approach  (Gray87]. 

3.5.  Summary 


Backward  error  recovery  is  well  suited  for  systems  in  which  error  recovery  is  complex.  Atomicity  is  more  eas¬ 
ily  achieved  and  error  recovery  is  context  free.  Code  modification  and  enhancement  are  also  simplified.  Unfor¬ 
tunately,  backward  error  recovery  introduces  overhead  which  degrades  normal  (error-free)  performance.  In 
addition,  the  process  of  recovery  can  remove  the  effects  of  previously  completed  work,  therefore  requiring  a 
method  of  reinstating  these  effects.  Furthermore,  communicating  processes  must  take  special  precautions  to 
avoid  the  domino  effect. 

4.  Approach 


Our  approach  to  error  recovery  is  to  pursue  the  advantages  of  backward  error  recovery  without  introducing 
overhe^  or  effecting  previously  completed  work.  It  is  based  upon  two  assumptions:  operations  do  not  guaran¬ 
tee  atomicity  and  operations  do  not  directly  communicate  with  one  another. 

In  the  remainder  of  this  section,  we  examine  the  details  of  this  approach.  We  begin  by  discussing  our  assump¬ 
tions.  Next,  we  present  our  approach  for  error  recovery  followed  by  a  description  of  the  error  recovery  mecha¬ 
nism.  We  then  examine  the  overhead  of  this  approach  and  conclude  with  a  discussion  of  our  ability  to  verify 
consistent  operation  of  the  system. 
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4.1.  Assumptions 


First,  we  assume  that  filesystems  or  databases  do  not  expect  storage  systems  to  guarantee  operational  atomic¬ 
ity.  We  believe  this  to  be  reasonable  because  all  storage  systems  available  today  can  fail  and  expose  partially 
complete  operations.  Given  freedom  from  atomicity,  we  can  recover  to  a  convenient  consistent  state,  other 
than  the  one  which  existed  prior  to  the  execution  of  the  operation  to  be  recovered  and  much  less  expensive  to 
reach. 

Second,  in  our  experience,  operations  in  a  disk  system  are  independent,  only  communicating  by  competing  for 
shared  resources.  This  absence  of  communication  allows  us  to  confine  error  recovery  to  the  operation  which 
encountered  the  error,  reducing  the  amount  of  work  undone  as  a  result  of  backward  error  recovery.  Further¬ 
more,  we  do  not  require  a  method  to  restore  completed  operations  because  only  the  failing  operation  is  recov¬ 
ered  and  it  is  not  complete. 


4.2.  Approach 


The  goal  of  the  error  recovery  process  is  twofold:  restore  the  system  to  a  consistent  state  and  successfully  com¬ 
plete  operations  which  encountered  an  error.  Our  approach  is  to  use  backward  error  recovery  to  remove  the 
effects  of  an  error,  then  move  the  system  to  a  convenient  consistent  state  and  complete  recovering  operations 
based  on  the  new  operating  state.  We  believe  this  to  be  the  proper  approach  for  two  fundamental  reasons.  First, 
by  always  beginning  operations  from  a  consistent  state,  we  greatly  reduce  the  number  of  paths  from  starting 
state  to  completion  state  which  must  be  constructed.  Second,  the  error  case  should  not  be  optimized  if  it  makes 
normal  execution  more  complex.  When  an  error  occurs,  consistent  operation  is  more  important  than  minor 
optimizations.  We  firmly  believe  this  to  be  the  proper  philosophy  in  highly-concurrent  systems  such  as  redun¬ 
dant  disk  arrays  in  which  error  recovery  is  a  complex  task  which  occurs  infrequently. 

When  an  error  is  encountered,  our  approach  requires  the  following  steps  be  taken: 

1 .  suspend  initiation  of  new  operations 

2.  allow  operations  already  initiated  to  either  complete  or  reach  an  error 

3.  release  the  resources  acquired  by  operations  which  encountered  an  error 

4.  reconfigure  the  system 

5.  using  a  new  strategy,  restart  operations  which  encountered  an  error 

6.  resume  initiation  of  new  operations 

In  order  to  transition  the  system  to  a  consistent  state,  global  state  will  need  to  be  modified.  This  is  easiest  when 
the  system  is  quiescent.  To  quiesce  the  system,  incoming  operations  are  queued  and  operations  in  the  middle  of 
execution  are  allowed  to  either  complete  successfully  or  execute  until  an  error  is  encountered.  Operations 
which  encounter  an  error  must  release  all  resources  which  they  have  acquired.  These  operations  are  neither 
complete  nor  failed  at  this  point,  but  are  simply  suspended  until  a  consistent  operating  state  has  been  estab¬ 
lished. 

When  the  system  has  reached  quiescence,  the  current  operating  state  can  be  reconciled  with  the  physical  state 
of  the  system.  Once  this  is  done,  operations  which  encountered  an  error  are  restarted  using  a  new  strategy, 
appropriate  for  the  current  operation  state.  It  is  important  to  understand  that  the  status  of  these  operations  dur¬ 
ing  error  recovery  remains  “execution  in  progress.”  The  initiation  of  new  operations  is  also  resumed  at  this 
time. 
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Figure  7  Antecedence  Graph  of  a  RAID  Level  5  Small-Write  Operation.  Given  a  block  of  dau,  this 
antecedence  graph  illustrates  the  ordering  dependencies  required  to  complete  a  RAID  level  5  small-write 
operation  as  described  in  Figure  3(a).  Actions  include;  allocation  of  memory;  reading  of  old  data  and  old 
parity;  writing  of  new  data;  exclusive-or  of  old  data,  old  parity,  and  new  data  to  generate  the  new  parity;  the 
writing  of  new  parity;  and  the  deallocation  of  memory.  Since  context  is  not  required,  disk  read  operations  (RD) 
do  not  indicate  the  logical  object  type,  data  or  parity,  being  read.  Because  of  this,  error  handlers  for  each  of 
these  actions  can  be  constructed  and  executed  independent  of  operation  type. 


Finally,  it  is  important  to  note  that  some  disk  systems  allow  clients  to  specify  the  relative  ordering  of  opera¬ 
tions  [ANSI91].  For  example,  some  filesystems  rely  upon  the  ordering  of  writes  to  prevent  filesystem  corrup¬ 
tion  [LefHer90].  This  ordering  must  be  preserved  throughout  error  recovery  process. 

43.  Mechanism 


The  recovery  mechanism  we  present  here  allows  operations  to  be  executed  to  increase  performance  during 
normal  operation.  Performance  is  increased  by  allowing  maximal  concurrency  of  actions  within  an  operation 
and  not  introducing  overhead  by  saving  recovery  data.  Also,  exploration  of  the  design  space  is  enabled  by  rep¬ 
resenting  operations  as  a  partially-ordered  set  of  actions.  We  begin  the  discussion  of  our  mechanism  by 
describing  this  representation. 

To  achieve  high  concurrency  during  normal  operation,  we  observe  that  operations  perform  a  specified  transfor¬ 
mation  of  state  and  can  be  implemented  as  a  partially-ordered  set  of  actions  which  collectively  perform  this 
transformation.  An  antecedence  graph,  a  directed  acyclic  graph  in  which  the  ordering  of  actions  composing  an 
operation  is  specified,  is  a  natural  way  to  represent  an  operation,  exposes  inherent  ordering  dependencies,  and 
allows  maximal  concurrency  to  be  achieved.  Figure  7  illustrates  such  a  graph  for  a  RAID  level  5  small-write 
operation. 

A  library  of  antecedence  graphs  is  constructed  from  a  pre-defined  set  of  actions,  such  as  those  found  in  Table 
1 .  When  an  operation  is  initiated  in  the  array,  a  graph  which  specifies  the  work  required  to  complete  the  opera¬ 
tion  is  select^  from  this  library.  The  criteria  for  graph  selection  includes  the  type  of  operation  requested  and 
the  current  operating  state.  Execution  of  the  graph  is  dataflow-like,  with  actions  executing  when  their  anteced¬ 
ents  have  completed.  By  requiring  that  all  actions  return  a  pass/fail  status,  the  task  of  satisfying  these  ordering 
dependencies  becomes  straightforward.  Obviously,  a  large  variety  of  graphs  can  be  constructed  from  a  small 
collection  of  actions.  Because  of  this,  designers  are  free  to  experiment  with  a  variety  of  strategies  by  simply 
constructing  new  graphs  and  adding  them  to  the  library. 

Recall  from  Figure  S  that  operations  which  detect  an  error  are  required  to  alter  their  strategy  to  reach  comple¬ 
tion.  Therefore,  the  antecedence  graph  currently  being  executed  must  be  terminated  and  replaced  with  a  differ¬ 
ent  graph  when  an  error  is  detected.  To  do  this,  forward  execution  of  the  current  antecedence  graph  for  this 
operation  must  be  suspended  and  the  resources  which  it  has  acquired  must  be  released.  This  is  easily  accom¬ 
plished  by  allowing  actions  which  have  already  begun  execution  to  complete  and  suspending  dispatch  of  fur¬ 
ther  actions  in  the  graph.  Once  the  in-flight  actions  have  completed,  we  work  backward  through  the  graph, 
releasing  resources  which  were  acquired  as  a  part  of  forward  execution.  This  processes  is  illustrated  in  Figure 
8  in  which  a  RAID  level  S  small-write  operation  has  encountered  an  error  while  attempting  to  write  new  data 
to  the  array. 


Page  13  of  IH 


Figure  8<a):  Error  Encountered  During  Forward  Execution 


Figure  8(b):  UNDO  Actions  Applied  During  Backward  Execution 


Figure  8  Graceftii  Terminatioii  of  a  Failed  RAID  Level  5  Smail-Write  Antecedence  Graph.  In  this 
illustration,  a  small-write  antecedence  graph  detects  an  error  while  attempting  to  write  new  data.  When  the 
error  is  detected,  execution  is  suspended,  meaning  that  the  XOR  operation  is  allowed  to  complete,  but  actions 
which  have  not  yet  been  initiated,  those  in  dashed  boxes,  are  not  allowed  to  begin.  Once  forward  execution  is 
tialted,  backward  execution  begins,  applying  the  UNDO  versions  of  the  actions,  found  in  Table  I.  When 
backward  execution  completes,  all  resources  allocated  by  the  graph  have  been  released. 


To  simplify  the  process  of  releasing  resources  we  define  for  every  action  a  corresponding  action  which  releases 
resources  which  were  acquired.  We  call  these  two  actions  DO  and  UNDO,  respectively.  Forward  motion 
through  an  antecedence  graph  executes  DO  actions  while  backward  motion  executes  UNDO  actions.  Table  1 
summarizes  the  actions  required  for  implementations  of  RAID  levels  discussed  in  Figure  1 . 

Error  handlers  are  constructed  for  each  error  status  that  an  action  might  return.  For  example,  a  read  of  a  disk 
(RD  in  Table  1 )  could  fail  due  to  parity  errors,  medium  failure,  or  timeout.  How  these  errors  are  handled  is 
arbitrary  as  long  as  they  are  handled  correctly.  For  example,  errors  which  result  from  the  inaccessibility  of  a 
disk  which  is  assumed  to  be  good  are  obligated  to  change  the  array’s  state  information  to  so  that  disk  is  viewed 
as  “inaccessible.”  By  doing  this,  subsequent  operations  will  not  attempt  to  read  the  inaccessible  device. 

Once  error  handlers  have  restored  the  system  to  a  consistent  operating  state,  new  graphs  are  selected  for  oper¬ 
ations  which  encountered  errors  and  are  submitted  for  execution.  These  graphs  implement  different  strategies 
to  complete  their  associated  operations,  based  upon  the  new  operating  state.  Also,  the  process  of  initiating  new 
operations  resumes. 


Table  1:  Actions  Required  to  Implement  RAID  Operations 


DO  Action 

Description 

UNDO  Action 

Description 

RD 

read  from  disk 

NOP 

no  operation 

WR 

write  to  disk 

NOP 

no  operation 

MEMa 

allocate  memory 

MEMo 

deallocate  memory 

MEMo 

deallocate  mem. 

NOP 

no  operation 

XOR 

exclusive-or 

NOP 

no  operation 

LOCKa 

acquire  lock 

LOCKr 

release  lock 

LOCKr 

release  lock 

NOP 

no  operation 
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4.4.  Overhead 


As  discussed  in  Section  3..  backward  error  recovery  intnxluces  overhead  in  ti*o  vnays  resources  are  rcc|uired 
to  hold  recovery  data  and  work  is  required  to  save  recovery  data  dunng  normal  pnKessinti  Our  approach  does 
not  introduce  overhead  since  no  additional  state  information  is  saved  as  a  part  of  normal  privessing  The  state 
information  we  must  restore,  resources  which  have  been  acquired,  is  already  known  The  method  used  lo 
release  these  resources  is  determined  via  a  table-lookup  during  error  recovery 

Additionally,  since  operations  do  not  communicate,  our  unit  of  recovery  is  an  operation  and  we  asoid  the  Jom 
ino  effect.  We  are  not  required  to  undo  previously  completed  operations.  Therefore,  a  log  of  completed  work 
does  not  need  to  be  maintained. 

Finally,  unlike  forward  error  recovery,  we  do  not  embed  code  throughout  the  forward  execution  path  to  iden¬ 
tify  the  state  of  the  system  at  the  completion  ot  each  action;  rather,  we  simply  assess  each  action  as  pass/fail 
and  then  continue  forward  or  backward. 

4  J.  Consistent  Operation 

By  specifying  an  operation,  its  antecedence  graph,  and  the  actions  in  the  graph,  we  can  reason  about  the  cor¬ 
rectness  of  an  operation.  This  is  accomplished  by  showing  a  correspondence  between  the  specification  of  an 
operation  and  its  implementation  which  is  represented  as  the  antecedence  graph. 

Consistent  operation  of  a  redundant  disk  array  requires  that  invariants,  specified  relationships  between  a  data 
object  and  the  redundancy  object  associated  with  it,  be  maintained.  Guaranteeing  that  invariants  are  main¬ 
tained  is  trivial  for  a  nondestructive  operation,  such  as  a  read,  which  alters  neither  data  nor  redundancy. 
Destructive  operations,  such  as  write,  are  obligated  to  modify  both  data  and  redundancy.  When  a  write  opera¬ 
tion  completes,  these  modifications  must  satisfy  invariants  between  data  and  parity. 

When  a  failure  occurs  during  a  write  operation  in  a  redundant  disk  array,  either  data  or  redundancy  will  be 
inaccessible.  The  surviving  data  or  redundancy  object  will  be  in  an  indeterminate,  but  accessible,  state  since 
the  error  may  have  occurred  either  before  or  after  its  update  was  performed.  Consistent  recovery,  therefore, 
requires  the  ability  to  overwrite  an  indeterminate  object,  making  it  correct.  This  process  of  resolving  determi- 
nacy  is  a  key  component  of  the  alternative  operation  strategies  of  a  retry. 


4.6.  Summary 


Our  approach  to  the  handling  of  errors  in  redundant  disk  arrays  is  based  upon  retry,  rather  than  continuation,  of 
operations  which  encounter  an  error.  To  simplify  our  approach,  we  make  two  assumptions  regarding  opera¬ 
tions:  they  do  not  guarantee  atomicity  and  they  do  not  communicate.  From  these  assumptions,  we  are  able  to 
construct  an  error  recovery  mechanism  which  does  not  introduce  overhead  during  normal  processing. 

When  an  error  is  encountered,  we  quiesce  the  system,  reconfigure  to  achieve  a  consistent  state,  and  retry  oper 
ations  which  encountered  an  error. 

Operations  are  represented  as  antecedence  graphs,  allowing  clear  reasoning  about  the  ordering  of  actions 
which  compose  an  operation  and  the  exploit  of  concurrency.  New  types  of  antecedence  graphs  are  easily  cre¬ 
ated  and  integrated  into  the  system,  greatly  simplifying  the  task  of  exploring  new  implementation  strategies. 

Finally,  by  specifying  state  transformations  of  operations,  antecedence  graphs,  and  actions,  we  can  demon¬ 
strate  correctness,  either  formally  or  informally. 
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5.  Coadnrioai  and  FMare  Work 


5.1.  CaockMloM 


By  RMkiaf  che  hawfimf  of  man  iMkpmdnM  of  (tie  cuturti  m  »hK-f«  ihr>  ucvur.  ««  aikm  cwlr  modulo  to  he 
more  easily  modified  This  makes  espkmioti  of  the  tlesign  sfMcc  easier,  alkming  desi|fscrs  of  redundani  disk 
amys  to  spend  more  lime  formulatinf  an  approach  ami  less  lime  implememin^  n 

By  simplifying  the  design  process,  we  enable  production  of  more  aggressive  RAID  algorithms  whKh.  ir. 
today's  environmeni.  are  arguably  too  compics 

By  using  aiuecedence  graphs  as  an  execution  model  for  an  operation,  we  expose  the  inherent  ordering  ot 
actions  which  compose  an  operation.  This  simplifies  the  scheduling  of  these  actions,  making  concurrenev  eas¬ 
ier  to  i^^>lefnem. 

Finally,  by  structuring  our  design  and  error  handling  process,  we  enable  verification  of  the  correctness  of  our 
design.  From  spectficattons  of  operations  and  error  handlers,  correctness  can  be  argued  either  formally  or 
informally. 

5.2.  Future  Work 


Work  is  in  progress  to  verify  our  approach.  We  ate  concentrating  on  three  efforts  to  validate  correctness,  per¬ 
formance,  and  complexity  r^uction.  First,  we  are  specifying  RAID  in  general  and  a  left-symmetric  implemen¬ 
tation  of  RAID  level  S  in  particular.  This  will  allow  us  to  argue  correctness.  Second,  we  are  implementing  a 
left-symmetric  RAID  level  5  driver  to  verify  performance  and  correct  operation.  Finally,  we  will  modify  this 
driver,  employing  more  aggressive  algorithms,  to  demonstrate  code  reusability,  the  ability  to  implement  more 
aggressive  RAID  technology,  and  the  ability  to  explore  the  design  space  by  simply  composing  new  operations 
from  an  existing  set  of  actions. 
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